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This article represents one outcome from the Invitational Research Symposium on 
Technology-Enabled and Universally Designed Assessments, which examined technology- 
enabled assessments (TEA) and universal design (UD) as they relate to students with 
disabilities (SWD). It was developed to stimulate research into TEAs designed to better 
understand the pathways to achievement for the full range of the student population 
through enhanced measurement capabilities offered by TEA. This paper presents impor- 
tant questions in four critical areas that need to be addressed by research efforts to 
enhance the measurement of cognition for students with disabilities: (a) better measure- 
ment of achievement for students with unique cognitive pathways to learning, (b) how 
interactive-dynamic assessments can assist investigations into learning progressions, 
(c) improvement of the validity of assessments for students previously in the margins, 
and (d) the potential consequences of TEA for students with disabilities. The current 
efforts for educational reform provide a unique window for action, and test designers 
are encouraged to take advantage of new opportunities to use TEA in ways that were not 
possible with paper and pencil tests. Symposium participants describe how technology- 
enabled assessments have the potential to provide more diagnostic information about 
students from various assessment sources about progress toward learning targets, gen- 
erate better information to guide instruction and identify areas of focus for professional 
development, and create assessments that are more inclusive and measure achievement 
with improved validity for all students, especially students with disabilities. 
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Overview 

The Invitational Research Symposium on Technology-Enabled and 
Universally Designed Assessments was held in Arlington, Virginia, on July 
23, 2009. Measured Progress and SRI International sponsored this meeting 
focused on the emerging and dynamic field of technology-enabled assess- 
ments (TEA) and the principles of universal design for assessment as they 
relate to students with disabilities. The symposium brought a group of 
researchers together from several areas of expertise including educational 
technology, cognitive psychology, students with disabilities, universal 
design for learning, and educational assessment. Among the participants 
were researchers who had completed or were engaged in research involving 
technology-enabled assessment, universal design for assessment, and/or 
students with disabilities, focused on two specific areas: cognition and 
access. The state of educational assessment and technology had recently 
been described in an article entitled Beyond the Bubble: Technology and the 
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Future of Student Assessment (Tucker, 2009). Tucker drew attention to 
assessment challenges in the context of a cognitive model for assessment, 
and envisioned a future for assessment and technology that resolved many 
of the challenges. Tucker’s article provided a foundation for the design 
of the symposium content, the meeting agenda, and motivated the plan 
to create a national research agenda that would capture the knowledge, 
expertise, and vision generated that day. 

To launch the symposium deliberations, four cutting-edge research 
initiatives were conveyed to symposium participants. Chris Camacho 
with Children’s Progress presented an adaptive and scaffolded assessment 
approach that provided prompts after incorrect responses and selected 
assessment items based on examinee responses to a previous question. 
Boo Murray, CAST, described an exemplar of universal design for learning, 
Strategic Reader, that assesses maze and oral reading fluency via a web- 
based tool. Jody Clarke, Harvard Graduate School of Education, described 
immersive virtual performance assessments under development and 
designed to assess knowledge and skills in science through items embedded 
within the context of virtual scenarios. Michael Russell of Nimble Tools 
demonstrated computer administered assessment tasks with embedded 
tools universally designed to facilitate access to content for students with 
special needs. 

The presentations demonstrating assessment and technology inno- 
vations were followed by a large-group dialogue and discussion between 
the presenters and participants about the research initiatives including 
unique challenges and particular innovations. This discussion delved into 
the target areas of cognition and access and surfaced insights and ques- 
tions arising from consideration of the future of TEAs. The debriefing ses- 
sion resulted in the large group dividing into two subgroups, one tackling 
issues regarding cognition and the other issues regarding access to assess- 
ment content. This article is based on the culmination of the symposium 
day plus the ongoing interactions among the subgroup members, who met 
to generate a research agenda regarding technology-enabled educational 
assessment and measurement of cognition for students with disabilities. 

Seventeen participants joined the symposium subgroup that addressed 
measurement of cognition and students with disabilities. The members of 
the cognition group included one researcher from a university, six from 
assessment publishers, seven from research institutes, one researcher 
from a national technology center, and two educational consultants. The 
subgroup members communicated via email and telephone conference 
calls over a nine-month period following the symposium. Fifteen of the 
participants were contributing authors, writing components of this article. 
Two members of the symposium planning team facilitated communica- 
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tion, assembled interim drafts, and assimilated revisions from subsequent 
reviews. Final edits were assembled by the facilitators and reviewed by 
contributing authors prior to submission for publications. 


Introduction 

States are rapidly incorporating technology in state achievement 
assessment programs. Use of technology to administer, score, and deliver 
assessment results, or technology-enabled assessment (TEA), not only 
provides the promise of greater efficiency, but more important, of pow- 
erful new capabilities to create assessments that better model good 
instruction and support more valid inferences about student proficiency 
(Tucker, 2009; Quellmalz & Pellegrino, 2009). 

Moreover, the forthcoming reauthorization of the Elementary and 
Secondary Education Act (ESEA), the emergence of common core standards, 
and additional federal funding to coordinate assessment efforts through 
state consortia provide a unique window for action (U.S. Department of 
Education, 2009). Increasingly, educators and policymakers demand more 
diagnostic information about students from various assessment sources 
about progress toward learning targets, data to inform targeted and suc- 
cessful instruction, and data to identify areas of focus for teacher and 
administrator professional development. The opportunity exists to use 
TEA in ways that were not possible with paper and pencil tests, to develop 
assessments that are more inclusive and also to create assessments that 
measure achievement with improved validity for all students, especially 
students with disabilities. 

As the nation embarks on an unprecedented redesign of its standards 
and assessments, at a time when the field of TEA is new, it is essential 
that students with disabilities be considered in research and development 
efforts at the outset of any endeavor. Otherwise it will become necessary to 
retrofit the new assessments for this population of students (Thompson, 
Thurlow, Quenemoen, & Lehr, 2002; Thurlow, 2009), which is an expen- 
sive, time-consuming, and less effective proposition. As researchers begin 
to explore the opportunities that TEA offers to improve measurement 
of achievement and access of all students, we call for a comprehensive 
national research agenda that focuses on students with disabilities as 
part of the larger assessment system. Important issues, questions, and 
problems need to be articulated to promote research that addresses how 
to ensure that technology-enabled assessments are appropriately acces- 
sible and are employed to increase the validity of the measurement of the 
knowledge, skills, and abilities (KSAs) of students with disabilities. 
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A comprehensive assessment system that is fully aligned with chal- 
lenging content standards is necessary for the acquisition of useful diag- 
nostic information that supports all students in achieving the standards. 
In this “ideal” comprehensive assessment system, a system of assessments 
(Table 1) would be designed to validly and reliably measure all of the 
content standards, not just those that can be most easily and efficiently 
measured (Herman, 2007). To accomplish this goal, assessments would 
incorporate multiple measures to reach the highest levels of cognitive 
complexity prescribed by the standards. Interim assessments would docu- 
ment student progress at key points in the school year to allow for instruc- 
tional corrections before the final summative assessment is administered 
(Perie, Marion, Gong, & Wurtzel, 2007). On-going formative assessment 
that is well aligned to learning progressions lead to achievement of the cul- 
minating standards. Learning progressions may be different for students 
with disabilities than for typical learners. Formative assessments would 
be continuously administered throughout instruction to make possible 
immediate instructional adjustments (Kingston & Nash, 2009). These 
assessments would be dynamic enough to chart all students’ learning, 
even if unique, and to drive instructional efforts efficiently toward stu- 
dents’ achievement of the content standards. Each component in a com- 
prehensive system would contribute to constant monitoring of student 
progress and continuous adjustments in the instructional delivery system 
(Sheinker & Redfield, 2001). All of the assessments in the system con- 
tribute to ongoing diagnostic information. 
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Table 1 : Components of an Assessment System 

Although there is no consensus on how to define and interrelate the components of 
a comprehensive assessment system, for purposes of the discussion that follows, the 
following definitions will be used: 


Formative 

Assessment 

As defined in the RFP (Race to the Top Fund, 2009), "formative 
assessment means assessment questions, tools, and processes 
that are embedded in instruction and are used by teachers and 
students to provide timely feedback for purposes of adjusting 
instruction to improve learning." 

Interim 

Assessment 

As defined in the RFP (Race to the Top Fund, 2009), "interim 
assessment means an assessment that is given at regular and 
specific intervals throughout the school year, is designed to 
evaluate student's knowledge and skills relative to a specific set of 
academic standards, and produces results that can be aggregated 
(e.g., by course, grade level, school, or LEA) in order to inform 
teachers and administrators at the student, classroom, school, and 
LEA levels." 

Summative 

Assessment 

As defined by Perie, Marion, and Gong (2009), "summative assess- 
ments are given at the end of instruction to provide information 
on what was learned. They are generally administered once a 
semester or year to measure students' performance against 
district or state content standards. Summative assessments 
are standardized, usually given statewide (but can be given 
districtwide) and are often part of an accountability system. 

While schools may use these data to identify students in need 
of extra support, they are not designed to provide teachers with 
timely information about their current students' learning." 


The actualization of a comprehensive system rests on a shift from cur- 
rent assessment practices. “Assessment practices should focus on making 
students’ thinking visible to themselves and others by drawing out their 
current understanding so that instructional strategies can be selected to 
support an appropriate course for future learning,” (National Research 
Council, 2001, pp. 90-91). Apart from typical accountability paradigms 
(and constraints), assessments purposed to provide instructionally rele- 
vant information should do more than target evaluations of current under- 
standing, knowledge, or traits. They should provide insight into specific 
reasoning strategies students use for problem solving and constructing 
meaning (cognitive pathways), as well as provide timely feedback on the 
students’ acquisition of knowledge and skills (learning progressions). Such 
assessment “...generally requires more complex tasks that reveal informa- 
tion about thinking patterns, reasoning strategies, and growth in under- 
standing over time” (NRC, 2001, pp. 62-63). 

This article has four sections. The first section reviews the potential 
and current uses of TEA to improve measurement of cognition of students 
with disabilities. The second section proposes considerations for designing 
a research agenda to address key issues that are especially pertinent to 
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students with disabilities. The third section poses questions for research 
related to the essential issues for students with disabilities, and the final 
section provides suggestions for designing and implementing the research 
agenda. 

Cognitive Pathways and 
Learning Progressions 

For the purposes of this paper, we will explore how cognition may 
impact student interactions with assessments and propose possible solu- 
tions for students with pathways that may differ from standard assump- 
tions about student thinking based on the impact of their disabilities. This 
paper proposes new thinking, but does so carefully because the notion 
of cognitive pathways is still relatively new. Currently, there does not yet 
exist widespread agreement regarding two key terms used throughout 
this paper: “cognitive pathways” and “learning progressions.” However, 
Pellegrino (2009, pp. 98-99) draws the following distinction. 

From a cognitive standpoint, development and learning are not the 
same thing. Some types of knowledge are universally acquired in the 
course of typical development, while other types are learned only 
with the intervention of deliberate teaching. 

In math, the concepts of ordinality and cardinality appear to develop 
in all non-infants without instruction. In contrast, however, such 
concepts as mathematical notation, algebra, and Cartesian graphing 
representations must be taught. 

So cognitive pathways apply to the growth or maturation of physio- 
logical/neurological structures and lead to skills that have not typically 
been the focus of classroom teaching or assessment. These structures 
impact most kinds of learning and include working memory, metacogni- 
tion, and fluid intelligence. Cognitive pathways, or developmental thinking 
processes, provide the platform for learning progressions, where learners 
build upon their existing declarative and procedural knowledge. 

Typically, a child must understand two-digit subtraction without bor- 
rowing before he/she learns two-digit subtraction with borrowing. Both 
are likely to be easier to learn if the child also understands place values. 
Together these three concepts form a small piece of a typical learning 
progression as we now understand it, which might include required and 
optional pathways. How (and how well) a child integrates these concepts 
will depend on the underlying cognitive structures and thinking skills 
available to that child. For example, a student with a smaller working 
memory capacity might require a different teaching approach and perhaps 
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a different assessment approach to determine whether the student has 
mastered the concept rather than whether he has remembered the proce- 
dure. 

Learning progressions — which are the content skill and concept 
building blocks to learning — may be only slightly less idiosyncratic than 
cognitive pathways. Heritage (2008) cites six descriptions of learning pro- 
gressions from recent research literature. All have in common a vertical 
progression within a domain, but differ in the implied size and breadth 
of each unit within the progression from Popham’s (2007) “carefully 
sequenced set of building blocks that students must master on route to 
a more distant curricular aim” to several authors who describe learning 
progressions as “descriptions of successively more sophisticated ways of 
thinking about an idea that follow one another as students learn” (Wilson 
8c Berenthal, 2005; Smith, Wiser, Anderson, 8c Krajcik 2006). In contrast 
to Popham’s definition, this latter description might encompass cognitive 
pathways as it incorporates thinking or cognitive complexity. Cognitive 
pathways and learning progressions may be intertwined. Cognitive path- 
ways allow learning to occur, and, as learning occurs, the cognitive path- 
ways are elaborated, thereby increasing capacity for traveling the learning 
progression. 

A critical question that needs to be investigated is, “Do learning pro- 
gressions differ for students with disabilities?” The presence of a disability 
may create the need for different cognitive pathways and varying learning 
progressions to content comprehension. For example, text comprehen- 
sion is often considered a necessary access skill for success in school and 
on assessments. However, students who are deaf do not grow up in a 
phonemic environment, so the cognitive pathways that provide the foun- 
dational architecture to support print literacy may differ from those of 
hearing students (Singleton, Supalla, Litchfield, 8c Schley, 1998). At the 
same time, students with learning disabilities (Edyburn, 2000) and stu- 
dents with visual impairments (Johnstone, Thurlow, Altman, Timmons, 
8c Kato, 2009) may learn to read print (or Braille) in similar ways to non- 
disabled students, but may choose to bypass visual reading and listen to 
text for comprehension in school. At the very least, students with disabili- 
ties for whom print creates a particular challenge may have different font, 
color, size, and color-overlay preferences that are impossible to reproduce 
on a paper-based test. In order to design and implement assessments 
that are accessible, adequate, and valid for use with students of varying 
abilities, it is important to understand the extent to which the cognitive 
pathways to learning vary across disabilities. This question is complex and 
requires attention to numerous considerations. 
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Potential of TEA 

The use of technology within the comprehensive assessment system 
offers opportunities to investigate the multiple pathways to learning and 
demonstrating knowledge that are unique for students with disabilities. 
In addition, experts recommend that assessments provide students with 
multiple representations of material and multiple means of response 
(Dolan, Hall, Banerjee, Chun, & Strangman, 2005). Such flexibility is dif- 
ficult with paper-and-pencil assessments and requires accommodations 
that are often inefficient, or expensive, or that may threaten the validity 
of the assessment. For example, having an adult paired with a student 
to read materials or capture responses may be problematic for a variety 
of reasons, such as introducing the possibility of changing the intended 
construct (Sirici & Pitoniak, 2007). Moreover, such approaches foster 
a dependency on others that is inconsistent with the educational goals 
for students. Technology-enabled assessments hold forth the promise of 
increased access (Almond et al., 2010) and supports that are efficient and 
effective and that maximize the capabilities of students with disabilities 
to demonstrate what they know and can do. Because students with dis- 
abilities are a heterogeneous group, the “alternate pathways to learning 
and assessment” argument laid out by the Center for Applied Special 
Technology (CAST) and provided in many online assessments may be a 
more appropriate way for assessing all students, especially this popula- 
tion. 

While still in early stages, there are several projects that exemplify 
work currently occurring to harness the potential of technology-enabled 
assessments to better measure achievement for all students. The implica- 
tions of these advances for a program of research pertinent to students 
with disabilities are described in this article. The projects described below 
attempt to reveal cognitive pathways and leverage learning progressions 
in new ways and at various levels to increase student access to the assess- 
ment and usefulness of findings from the assessment for mapping ways to 
increase learning by informing teaching. 

Oregon's Computer-adaptive Assessment (CAT) 

CATs have been used for about 30 years, but not for summative assess- 
ments for accountability until recently accepted for use in Oregon. Oregon 
State Department of Education developed a within grade level summative 
adaptive assessment (http://www.ode.state.or.us/testing/manuals/2007/ 
doc4.2adaptivealgorithm.pdf). CATs save testing time by giving each stu- 
dent only those items likely to yield useful information. Typically, adap- 
tations are determined based on the average difficulty of items across 
the entire population of test takers. For example, very easy items are not 
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presented to students for whom previous responses indicate the items 
will surely be responded to correctly. The promise of adaptive testing to 
measure achievement as currently designed, however, depends on two 
underlying assumptions: (1) all students are taught in the same scope 
and sequence and, (2) learning progressions within each content area are 
largely common across students (Ash, 2008, p. 21). Students must have 
maximum opportunity to demonstrate knowledge at grade level before 
assessments are “adapted” and focus shifts to different or less complex 
constructs. Without these assumptions being met, the ordering of items 
according to difficulty will be different for different groups of students, 
particularly students with atypical learning styles or abilities. Results may 
be misleading when test takers respond to questions in unexpected or idio- 
syncratic ways (Cahalan-Laitusis, 2009). Given these caveats, however, the 
potential exists to explore students’ learning in a manner different from 
paper and pencil tests, which typically concentrate items at the proficient/ 
not proficient cut points. 

Children's Progress Academic Assessment 

(http://www.childrensprogress.com/ products/benefits-of-cpaa.shtml.) 

This assessment incorporates adaptive functions with the addition 
of scaffolding (hints or prompts that increase student access to the con- 
struct) in a formative assessment, to provide children with feedback and 
support. The program covers concepts in early literacy and mathematics. 
The theoretical foundation of the design is to identify the Zone of Proximal 
Development, which is defined by Vygotsky as “the distance between the 
actual development level as determined by independent problem solving 
and the level of potential development as determined through problem 
solving under adult guidance or in collaboration with more capable peers” 
(Vygotsky, 1978). One of the principles behind the assessment is that as 
much information can be gained from a child’s incorrect responses as from 
correct responses. Adaptivity rules are based on classical psychometrics as 
well as the conceptual connections between items as described by content 
experts; learning progressions are considered along with item difficulty 
indicators. Scaffolding, through hints and prompts in this assessment, is 
based on distracter analysis that identifies students’ common misunder- 
standings. The goal is to directly tie assessment with instruction and pro- 
vide instructional recommendations for teachers based on student results. 

Harvard University Immersive Virtual Performance Assessment 

Another example of the innovations available through technology- 
enabled assessment is the development of virtual performance assess- 
ments by Harvard University (http:/ /virtualassessment.org/). The Harvard 
project aims to summatively assess achievement in scientific inquiry based 
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on research in Evidence Centered Design proposed by Mislevy, Steinberg, 
and Almond (2003). The student experiences an immersive virtual per- 
formance assessment via computer interface, with 3-D virtual contexts, 
digital artifacts, and avatar-based identities that capture what students 
are doing. It has the look and feel of a video game, but is based on an 
authentic setting allowing a realistic causal model for experimentation. 
Everything the student does is captured in the database, and this capture 
allows real-time analyses of the student’s path toward solving the problem, 
comparing student pathways to pathways that would be used by experts in 
this situation. 

CAST Strategic Reader 

There are also important efforts underway to develop computer- 
enabled assessments specifically for classroom applications for students 
with disabilities. For example, CAST has developed Strategic Reader, a 
web-based research prototype reading environment (http://cast.org/ 
research/projects/pm.html) that includes curriculum based measurement 
tool, which can be used for Response to Intervention (Rtl) assessments. 
Two curriculum based measures have been embedded — maze and oral 
reading fluency. The maze test is immediately scored and results reported 
in various formats. The oral reading fluency measure is recorded audibly, 
so teachers can listen to the audio and score later. The computer does all 
calculation of the scores and produces various displays that allows teachers 
to view student progress and determine when and which interventions are 
needed. 

Current Status of TEA 

Evidence of state reform efforts to incorporate technology into stan- 
dards and assessments has been demonstrated in a recent National Center 
for Educational Statistics (2009) report. By 2008-09, all 50 states had 
standards for students that included technology, 29 states had estab- 
lished a virtual school, 26 states offered online large-scale assessments, 13 
states tested students on technology standards, and 3 states required that 
teachers and administrators be trained in the use of technology. While 
more than half of the states now employ large-scale online assessments, 
the use of technology for assessments is proceeding more slowly than for 
instruction and has not resulted in changes to traditional approaches to 
testing. Most of the online tests used by state programs are replications 
of these traditional paper and pencil versions consisting primarily of mul- 
tiple choice questions. Only a few include essays, and even fewer use com- 
puter-adaptive testing, which has been used for years for certification and 
admissions testing. 
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Reasons for using technology are multiple: (1) technology is at the core 
of virtually every aspect of our daily lives and work, and we must leverage 
it to provide engaging and powerful assessments that measure student 
achievement in more complete, authentic, and meaningful ways; (2) tech- 
nology-based assessment systems will be pivotal in improving student 
learning and generating data that can be used to continuously improve 
the education system at all levels (Office of Educational Technology, 2010, 
p. 3); (3) many students come to school as “digital natives” who have spent 
their entire lives using the toys and tools of the digital age, and because of 
this exposure, they are starting to think and process information differ- 
ently (Prensky, 2001); and (4) technology provides the potential to trans- 
form the daily educational experience much as it is transforming other 
parts of society. Technologies gaining use in classrooms include not only 
desktop and laptop computers, but digital whiteboards, cell phones and 
smart phones, handheld or mini computers, clickers, Alphasmart key- 
boards, LCD projectors, USB drives, portable media players, and digital 
cameras (Simba Information, 2009), although schools continue to face 
struggles with the barriers of cost and access to hardware, software, and 
infrastructure. 

Considerations for Designing a Research Agenda 

In defining the key issues and suggesting research designs, the authors 
considered several priorities for building an evidence base for technology- 
enabled assessments, specifically tailored for students with disabilities: 

• How can we build a theoretical and empirical base for a new 
generation of assessments that allow us to measure and foster 
skills that cannot currently be measured well for all students? 

• What would a comprehensive assessment system that builds on 
the opportunities that TEAs provide look like? And how might 
the assessment tools within the system using TEAs provide 
more valid, complete, and actionable information for teaching 
and learning than our current assessments, which typically 
indicate what students with disabilities do not know or can not 
do? 

• How do we design the research to gather the evidence needed to 
help policymakers answer questions about efficacy, efficiency, 
validity, and value-added characteristics of technology-enabled 
assessments for students with disabilities? And how do we 

do this and implement it within a timeframe that is useful to 
practitioners and serves the pressures faced by policymakers? 
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Key Issues and Assumptions for Framing a 
Research Agenda 

Better Understanding of Cognition 

Technology may provide an opportunity to increase our depth of under- 
standing about cognition and the variance that likely exists in how stu- 
dents with a diversity of disabilities progress toward learning outcomes. In 
building on what is currently known about how students achieve mastery 
in reading, mathematics, and science, we can anticipate that investiga- 
tions of how uniform or varied the ways in which students progress from 
novice to expert may be increasingly possible using technology-enabled 
assessments. For example, can students with disabilities that affect lan- 
guage or reading fluency approach mathematics problems through graphi- 
cally based problem-solving strategies to reach the same understanding 
as typical learners? Or can investigations clarify whether, as Mislevy 
contends, “Rather than seeing such variables as independently existing 
characteristics of people, we can view them as summaries of patterns in 
situated behaviors” (Mislevy, 2009, p.l) that make for similarities in what 
students do on assessments. What can we learn about the strategies that 
students use to understand text and draw inferences across passages, for 
example, when working memory or executive functioning skills may be an 
issue? Technology-enabled assessment may provide an increased oppor- 
tunity to capture the complexity of competent performance necessary to 
design assessments from which we can make more valid inferences about 
the achievement of all students (Pellegrino, 2004; Tucker, 2009). 

In order to fulfill the potential of TEA to provide greater accuracy in 
determining what students, especially students with disabilities, know 
and have mastered, the authors focused on the following issues in their 
discussions: 

• Information derived from TEA is meaningful to the instructional 
decisions teachers need to make. 

• Teachers use the information gained from these assessments to 
guide instruction. 

• These assessments contribute to improvements in achievement, 
especially the higher levels of cognitive complexity that students 
may achieve. 

• Students, especially students with disabilities, have access to all 
assessments in the system and be fully engaged by the items and 
tasks. 
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A dilemma for measuring achievement of students with disabilities 
is the uncertainty about potential barriers that the assessments’ struc- 
ture and format might present to an accurate determination of what the 
student knows and can do. Technology offers flexible tools and formats 
to increase access. For example, font size and color can be individually 
adjusted for optimum visual access. The possibilities have not been fully 
investigated and may not yet be known as the full potential of technology 
unfolds. (See the article in this issue, Technology-Enabled and Universally 
Designed Assessment: Considering Access in Measuring the Achievement 
of Students with Disabilities — A Foundation for Research [Almond et al., 
2010], for a more in-depth discussion of this topic and a future research 
agenda focused on access for students with disabilities.) 

More Rigorous Standards 

States have struggled to measure more rigorous standards using 
current assessment methodologies. States and national organizations 
debate the importance of assessing skills beyond basic academics that 
meet international benchmarks (NGA, CCSSO, 8c Achieve, 2008; Jerald, 
2009). The Council of Chief State School Officers (CCSSO) recognized that 
discrepancies existed in the degree of challenge evident in state assess- 
ments resulting from differences among state standards. CCSSO and the 
National Governors Association Center for Best Practices (NGA Center) 
partnered with Achieve, Inc., ACT, and the College Board to initiate a pro- 
cess to develop a common core of state standards. This common core is 
intended to prepare students to compete nationally and internationally. 
It is still unclear what impact these standards will have on state assess- 
ments, and it is likely to be some time before the outcome is apparent. At 
present, states may not be currently assessing the most challenging stan- 
dards that already exist in their policy documents. For example, reading 
standards in many states address skills like critiquing and interacting with 
various forms of text (Johnstone, Thurlow, Thompson, 8c Clapper, 2008), 
but may not assess these areas on statewide tests (Johnstone et al., 2007; 
Johnstone 8c Thurlow, in press). Efficiency and costs may limit opportu- 
nities to accomplish this goal. TEAs may open doors for the assessment 
of already existing high standards or the adoption of more rigorous stan- 
dards, perhaps even common core standards, where the limitations of 
traditional assessments previously excluded them (Tucker, 2009). The 
innovative approaches to assessment that TEAs make possible go beyond 
the limited representations of learning shown on traditional assessments 
and provide a window into cognition that traditional assessments cannot. 
This innovation is especially critical for students whose disabilities have 
prohibited them from demonstrating what they know and can do on tradi- 
tional paper and pencil tests. 
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Improved Validity 

The utility and expanded use of TEA hinges on the establishment 
of evidence of validity. Establishing the validity of these assessments is 
necessary for their use in summative assessments for accountability and 
establishing a relationship between formative assessments and summa- 
tive outcomes. Technology’s flexibility, range, and ability to collect and 
combine multiple sources of data offer the promise of bridging the often- 
perceived distance between learning and assessment. Evidence of validity 
will be necessary to defend TEAs against arguments for comparability to 
the status quo. 

An assumption here is that comparability is not necessarily evidence 
of validity. Innovations in assessment that increase accuracy of results 
and open windows into the varying cognitive pathways that individual 
students exhibit should not be held hostage to “comparability” with tradi- 
tional forms of assessment. Technology-enabled assessments provide the 
opportunity to gather more information about the cognitive pathways and 
learning progressions individual students follow in reaching full achieve- 
ment of the standards. Rather than achieving comparability between two 
significantly different forms of assessment, the emphasis for the technical 
quality of any assessment should be increased validity for the intended 
purpose (AERA, APA & NCME, 1999). 

The No Child Left Behind Act Peer Review Guidance calls for evidence 
of comparability (NCLB, 2004). Comparability between TEAs and paper 
and pencil assessments and the results they produce may cease to be an 
issue if: 

• The TEA provides increased access for students with disabilities. 

• The increased depth and range measured by the assessment 
becomes the primary source of any incomparability. 

• Evidence of increased validity of technology-enabled assessment 
becomes the source of incomparability because the TEAs provide 
more accurate evidence of what students, especially students 
with disabilities, know and can do compared to paper and pencil 
tests. 

• The source of any incomparability of results across years in 
the transition from paper form to technology platform is 
determined to be the result of improvement of the inferences 
that can be derived from TEA. 
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Whether or not the upcoming reauthorization of ESEA redefines 
assessment or accountability parameters, best practice in assessment 
requires pursuit of improvement in the validity of assessment results, 
the inferences drawn from it, and the usefulness of that information for 
improving student learning. 

Evidence of Value 

Finally, only when adequate infrastructure is present in schools can 
TEAs be fully implemented. Motivating policymakers to provide the 
needed funding for adequate infrastructure requires convincing evidence 
of the efficacy, efficiency, validity, and value-added possibilities that tech- 
nology-enabled assessment offers. Research on TEAs thus far has dem- 
onstrated their potential to assess students in ways that provide a deeper 
understanding of student learning and that provide information that 
cannot otherwise be acquired in other formats (Clarke-Midura, Dede, & 
Mayrath, 2010). More evidence is needed to demonstrate and document 
these attributes, including research that substantiates the capacity of 
TEAs to improve teaching and learning of students with and without dis- 
abilities by providing valid, accessible, and practical assessments of cogni- 
tively challenging content and skills. 

One example of how TEAs can make possible an assessment that can 
provide a deeper understanding of student learning, particularly of more 
complex content applications, is the immersive virtual assessments cur- 
rently being developed and researched at the Harvard Graduate School 
of Education. This research project promises to make possible the assess- 
ment of science inquiry skills, an area of interest to content teachers and 
assessment designers that has proven difficult if not impossible to mea- 
sure on conventional assessments (Clarke-Midura et al., 2010). This type 
of assessment may enhance the ability of students with disabilities to 
demonstrate more complex learning when more visual information that 
relies less on text and more on interactivity, such as “drag and drop” and 
audio instruction, is available so that communication of the task is not 
totally or mostly dependent on or confounded by language. Harvard’s 
Graduate School of Education’s immersive virtual performance assess- 
ments provide strong visual images made uniquely possible via TEA to 
measure students’ science inquiry skills. Figure 1 displays an image from 
River City, an immersive virtual assessment of middle school science that 
takes students through a visually rich virtual simulation of environments. 
River City presents students with a problem and asks them to develop a 
hypothesis and procedure, virtually test their hypothesis, describe their 
findings, and make recommendations. 
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Figure 1 : Harvard Graduate School of Education's immersive virtual 

performance assessments provide strong visual images made 
uniquely possible via TEA to measure students' science inquiry 



Still, there are a number of important research questions to think 
about as we discuss this topic focusing on students with disabilities. The 
research questions focus investigations around the following key questions: 
(1) How do cognitive pathways and learning progressions vary across dis- 
abilities, and how can technology-administered tests take advantage of 
this knowledge? (2) How does the use of technology impact the ability 
to measure the constructs of interest, especially for students with dis- 
abilities? (3) What is the impact of using technology to administer tests, 
including performance assessments, on item creation, item type, and 
analysis of assessment results? (4) Which data matter for instruction and 
for score interpretation? and (5) How do we model the data to be useful 
for various purposes? 

Designing Measures of Learning Progressions 

Simply stated, learning and development are not the same thing. 
Cognitive pathways are physiological structures. Learning progressions 
are building blocks that students master on route to a learning outcome. 
Evidence-centered design (ECD) offers an approach to assessment design 
that incorporates attention to both cognitive pathways and learning pro- 
gressions. This assessment design framework is used to increase con- 
struct validity by executing a rigorous procedure consisting of five layers: 
domain analysis, domain modeling, conceptual assessment framework, 
and compilation, and the fifth layer, which is a four-phase delivery archi- 
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tecture (Almond, Steinberg, & Mislevy, 2002; Mislevy et al., 2003; Mislevy 
& Haertel, 2006). The result is an assessment argument that is able to 
link theories of cognition to student performances as evidence. In using 
ECD to design an assessment of a learning progression, the assessment 
designers conduct a domain analysis to identify the cognitive constructs 
that are of interest in the domain. During the domain modeling phase, the 
knowledge, skills, and abilities (KSAs) to be assessed are further specified 
to reflect a particular learning progression of interest. The ECD framework 
requires designers to create design patterns and task templates. These doc- 
uments link the knowledge and skills being assessed to the kinds of evi- 
dence that will reveal that students have attained the knowledge and skills 
associated with the learning progression and may hold implications about 
their cognitive pathways. 

Design patterns created during the domain modeling phase and task 
templates can be used for the design of multiple items/tasks. As part of 
the framework, designers identify the features of items/tasks that can be 
varied to detect different levels of performance of the learning progres- 
sion or pathway. For example, a variable feature may be the type of perfor- 
mance a student is expected to provide (an explanation vs. recalling a fact), 
the type of data display used (histogram vs. representation such as scat- 
terplot), or the number of data points in a graph. TEA offers the oppor- 
tunity to customize items/tasks at delivery, when specific items/tasks are 
chosen for given students. While it is important to keep the end in mind, 
for example, achievement targets expressed in achievement-level descrip- 
tors, designers have found that TEAs can provide a wealth of information 
about the varying cognitive pathways and learning progressions students 
follow toward these achievement targets. Working backward from the ulti- 
mate goal as described by the achievement-level descriptors, designers can 
map significant points along the content-specific learning progression and 
escalations in cognitive complexity leading to the goal. 

Other assessment designers have discussed at length the need to attend 
to the cognitive load of assessments by creating collections of items that 
reach the highest levels of cognitive complexity, but also map the cognitive 
pathway (Burkhardt, 2009; Schoenfeld, 2009). TEA provides an opportu- 
nity to increase understanding about what students with disabilities know 
and can do and how they know it. By deconstructing the cognitive fea- 
tures of content KSAs with attention to the cognitive complexity of what 
is being required, designers of TEA can increase the accuracy with which 
the nature of mastery for individual students is measured relative to iden- 
tified learning outcomes. This information is useful for skilled instruction 
as well as accurate measurement of degree of mastery of content concepts. 
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Assessments that take cognition into account rely on detailed 
models of the goals and processes involved in mental performance ... 
(NRC, 2001, p. 63).” The information about cognitive pathways needed for 
such design efforts is only now being acquired. Assessment designs that 
allow researchers to investigate the cognitive pathways and learning pro- 
gressions of individuals and groups of students have been largely unavail- 
able. Models like the Children’s Progress Academic Assessment and the 
ECD framework provide some clues to how TEA might open the door to 
the design of assessments that reveal these unique aspects of cognition, 
learning, and performance. 

Measuring the Construct of Interest Using TEA 

Scientific knowledge about computer-based tools, sound assessment 
design, student access to demonstrating performance, and cognition 
should be the foundation for building technology-enabled assessments. 
Building upon such a foundation is likely to enhance the validity or trust 
in and utility of the assessment results. Haladyna and Downing (2004) 
note that “the most fundamental step in validation is defining the con- 
struct” (p. 25). Evidence of the application of sound assessment design 
per accepted design principles, such as those described by Downing and 
Haladyna (2006), and evidence-centered design (ECD) (Mislevy et al., 
2003; Mislevy & Haertel, 2006) should be considered in both the design 
of and research on TEAs, particularly in regards to investigating validity 
evidence to demonstrate that the assessment is measuring the construct 
of interest (Clarke-Midura et al., 2010). 

Bejar et al. (2003) noted in their discussion of Embretson (1983) that 
“construct representation [is] a key aspect of test validity concerned with 
understanding the cognitive mechanisms related to the item solution and 
item features that call on these mechanisms” (p. 4). This concept is true for 
all types of assessments. In the context of TEAs and their environments, 
clear articulation of the measurement construct is key to understanding 
the role of technology in the measure, the potential benefits of technology, 
and its potential pitfalls. 

The interfaces, tools, modes of presentation of the content, and modes 
of response are item and assessment features with the potential to impact 
student performance. If performance is impacted because the feature is 
relevant to the construct being measured (construct-relevant), then it 
is functioning appropriately; otherwise, the feature is construct-irrele- 
vant and is evidence against validity of the assessment. For example, if 
an assessment requires a student to read and understand an unfamiliar 
vocabulary word before he/she can demonstrate understanding of the 
attributes of a main character in a narrative passage, the word itself may 
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be irrelevant to and an obstacle to the student’s ability to explain what he/ 
she learned about the character from the entire passage. 

Questions researchers hope to answer are whether cognitive load is 
increased, and if so, by how much, and is this additional cognitive load a 
feature of the construct being measured or external to the construct (Paas, 
Renkl, & Sweller, 2006). For example, in the work of Clarke-Midura et al. 
(2010), does the use of an avatar introduce a construct-irrelevant fea- 
ture that impacts the validity of results or the inferences about construct 
knowledge that can be made? The impact of features like this that are pos- 
sible in TEA is not as well researched or understood as the impact of fea- 
tures of traditional paper-based assessments. It is, therefore, important 
that future research investigates the impacts that technology environ- 
ments have on student performance to understand what features relate 
to or that might interfere with the validity of the assessment, especially 
those which may be affected by a student’s disability, and, just as impor- 
tant, what can be done to minimize construct irrelevant impacts. 

However, for all of these implementations of technology in assess- 
ment, the benefits of each “enhancement” must be weighed against the 
potential negative impacts. The test item creation process requires con- 
tinual investigation of how students think about and respond to specific 
test item features (Leighton & Gokiert, 2005). Although extensive research 
has been performed to examine the statistical properties of test items, less 
effort has been directed at understanding the potential for item features 
to introduce or reduce construct-irrelevant variance (Ferrara et al., 2003; 
Haladyna & Downing, 2004; Leighton & Gokiert, 2005; Gorin, 2007). 

Therefore, cognitive models for how students will interact with the 
assessment items, given their unique cognitive pathways, and the tech- 
nological features of TEAs must be posited and investigated. These cogni- 
tive models should not only be statistical in nature. They should include 
both the problem-solving strategies that students use within specific aca- 
demic content domains and those they use for navigating and using the 
technology. Thorough investigation of these models across student popu- 
lations will help to identify irrelevant and confounding elements within 
the assessment, provide greater validity through better understanding of 
the knowledge and processes students use to respond to assessment tasks, 
provide clarity about the needs for data capture during live assessment, 
identify potential mechanisms for using and understanding complex data, 
and identify how students can best demonstrate their learning. 

Understanding access and cognition in light of the measured domain- 
specific constructs within a TEA should provide improvements in 

(1) the assessment’s ability to measure what students actually know; 

(2) the assessment’s ability to measure how students use the knowledge, 
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understanding and skills in new and different contexts; (3) students’ 
abilities to navigate and respond within the assessment environment — a 
critical element for students with physical and cognitive disabilities; and 
(4) instructionally useful information about student thinking, areas 
needing improvement and extension, and strategies to close learning gaps, 
correct misunderstandings or enhance learning progress by mapping next 
steps to increased complexity. 

Further, the current psychometrics and test designs consistently yield 
relatively low levels of precision or high levels of measurement error for 
students at the “extremes” of performance (e.g., lowest and highest levels). 
The lack of precision at the lowest performance levels particularly affects 
students with disabilities, and yields low validity, trust, or utility of such 
scores. When students either top out or bottom out, the assessment is 
likely not efficiently measuring students’ actual knowledge or providing 
useful information for extending that knowledge, even for top performers. 
Computer-based assessment tools could adapt to, and extend, such per- 
formance opportunities at the extremes, thereby adding variability in 
scores, and increasing reliability and validity. Such computer-based assess- 
ment tools might include the use of adaptive algorithms for item selec- 
tion (CAT), incrementally providing hints or supports (scaffolding), and 
continuous feedback, simulated environments, and other approaches not 
yet imagined. 

Measuring Incomplete Understandings Using TEA 

Assessments, if they are to be instructionally supportive, should be built 
in ways that reveal the cognitive strategies students utilize in accessing and 
responding to test items, including both correct and incorrect responses, 
so that both remediation and enrichment instruction can be appropriately 
and successfully applied. As Pellegrino and his collaborators note, there is 
“value [in] describing students’ incomplete understandings” (NRC, 2001, 
p. 84). As noted before, TEAs make possible the opportunity to gather a 
wealth of information about these cognitive strategies. When technology 
is employed, opportunities increase to make assessments more interactive 
and dynamic than they might be otherwise, so design of the assessment 
becomes critical. In this section, we discuss interactive, dynamic, and scaf- 
folded assessment. The following terms are used to discuss these topics in 
this section. 

The terms “interactive” and “dynamic” are used together and sepa- 
rately in the literature. The definition of interactive-dynamic assessment 
comes from the field of intelligence testing and the dissatisfaction with 
conventional methods to provide information about individual’s learning 
that can be translated directly into practice by educators and members 
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of other helping professions. An important characteristic of interactive- 
dynamic assessment is identification of specific obstacles that may be hin- 
dering cognitive performance and specification of conditions under which 
intellectual performance can be facilitated (Tzuriel & Haywood, 1992, pp. 
8-9). For the purpose of this article, we discuss several types of interac- 
tive assessments and assessment strategies made possible through tech- 
nology. Interactive assessments can provide the opportunity for feedback 
to students and teachers during the assessment process. 

Table 2: Interactive Assessment 

Interactive assessment is a type of TEA that can impact the flow of instruction and is 
responsive to student performance (Byers, 2001). 


Dynamic 

Assessment 

Dynamic assessment is a type of interactive assessment based 
on Vygotsky's Zone of Proximal Development that integrates 
instruction and assessment (Poehner, 2008). Dynamic assessment 
provides corrective feedback in response to student failure to 
measure both the product and process of learning (Caffrey et al., 
2008). 

Scaffolding 

Scaffolding is a dynamic assessment strategy used to enhance 
items derived from supports provided during learning that are 
gradually removed when learning becomes solidified and/or the 
learner becomes more independent. Scaffolding in assessment 
includes structural assistance introduced to organize information 
or guide responses embedded in the presentation of the item 
or task (Perie, 2009, p. 377). Scaffolding is expected to affect the 
knowledge, skills, and abilities (KSAs) required to respond to a 
task, but is not intended to change the construct being measured. 
More information may be obtained regarding student cognition 
than in static assessments when only partial understanding of the 
construct is evident. 

Computerized 
Adaptive 
Testing (CAT) 

Computerized adaptive testing (CAT) is a type of interactive 
assessment, which redesigns educational measuring instruments 
for delivery by interactive computers. Its objective is to select, for 
each examinee, the set of test questions from a precalibrated item 
bank that simultaneously most effectively and efficiently mea- 
sures that person on the trait (Thompson & Weiss, 2009). 


Dynamic Assessment 

Dynamic assessments have been shown to have significant benefits 
for assessment and instruction (Grigorenko, 2009). A critical feature of 
a dynamic assessment approach is to link assessment with instruction 
effectively by providing actionable information to support instructional 
adjustments (Black 8c Wiliam, 1998). Although there are many different 
approaches that fall under the umbrella of dynamic assessment, there are a 
few key aspects that are consistent. Generally, dynamic assessments quan- 
tify a child’s learning potential, whereas static assessments typically gauge 
a child’s state of preexisting knowledge. The primary difference between 
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the two approaches lies in the fact that dynamic assessments adopt a 
Vygotskian approach to assessment. That is, dynamic assessments typi- 
cally provide various types of scaffolding after incorrect responses to dis- 
sociate what a child can do independently versus what a child can do when 
provided with scaffolding. Through the scaffolding procedures, dynamic 
assessments can shed light on particular misunderstandings that may be 
responsible for a child’s incorrect response. On the other hand, because 
static assessment does not provide scaffolding after incorrect responses, 
it is able to reveal only two states of understanding, unaided success and 
unaided failure (Fuchs et al., 2008). For the purposes of accountability 
over the past several years, static assessments seemed useful. In these 
assessments, students either knew or did not know grade-level material. 
Such assessments were useful for broadly understanding student success 
on grade-level standards, but have been less useful for instructional deci- 
sion making. 

In terms of instructional utility, dynamic assessment has been named 
as a promising alternative to static assessments for students with spe- 
cial needs, because this type of assessment may be able to determine 
the adequacy of students’ responses to interventions (Fuchs 8c Fuchs, 
2008; Fuchs, Fuchs, McMaster, 8t Al-Otaiba, 2003). Specifically, research 
has shown that dynamic assessments may predict an individual’s poten- 
tial to learn better than static measures (Grigorenko 8c Sternberg, 1998; 
Sternberg 8c Grigorenko, 2002). Dynamic assessment is able to account 
for variations in performance that may be due to factors such as prior 
instruction or a misunderstanding of the task directions, which cannot 
be accounted for by static testing (Samuels, Schermer, 8c Reinking, 1992). 
Therefore, it might be possible to use dynamic assessment to better pre- 
dict whether the student is likely to respond to instructional interventions 
or whether a more intensive intervention should be prescribed earlier on 
in the intervention process. 

Scaffolding 

A primary question for dynamic assessment is: How does the child 
respond to the feedback and scaffolding procedures? It is this reaction 
to feedback and scaffolding that dynamic assessments attempt to quan- 
tify. Scaffolded assessments logically include items to allow for gradually 
increasing the difficulty or cognitive complexity of the items/ steps related 
to the concept to be measured and hopefully to lead to a truer measure 
of the students’ level of performance or achievement. Scaffolded items 
may be developed to logically build meaning when complex concepts are 
embedded and contain first steps/questions that address underlying/ 
background knowledge and skills, while subsequent steps might more 
closely approach the content standards intended for the grade level being 
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assessed. For students who seem to be performing above or below the 
desired achievement level, scaffolded items may allow for assessing con- 
tent above and below the complexity/ difficulty level of the grade level con- 
tent standard or within agreed upon “bands” of difficulty within a grade 
level. 

Scaffolded items may also deconstruct the complexity of the con- 
struct being measured to allow students to show partial understanding of 
complex KSAs. A project currently in progress for a General Supervision 
Enhancement Grant for Montana is investigating the effects of scaffolding 
strategies in an alternate assessment based on modified achievement stan- 
dards. Four scaffolding strategies have been developed for both reading 
and mathematics assessments, which are designed to utilize instructional 
prompts teachers typically employ to redirect students from common mis- 
conceptions to the targeted KSA (Bechard, 2010). 

Through the use of scaffolded items, students can presumably better 
direct their own learning because they know what they know and what 
they do not yet understand, and teachers can likely better tailor instruc- 
tion to meet student-learning needs. By providing feedback and scaf- 
folding, dynamic assessment makes the evaluation a bidirectional process 
intended to mimic the child’s daily learning environment. 

Identifying Instructionally Useful Error Patterns Using TEA 

Recent research, test development, and score reporting have paid little 
attention to the identification of errors or error patterns for examinees, 
either as a whole or for subgroups. Even for traditional (i.e., summative, 
paper-pencil, static) assessments, investigations of errors can readily be 
conducted to provide instructional information, as well as assessment 
design and item development improvements. As assessments become 
more technologically enabled, innovations in item types and test admin- 
istrations, such as scaffolded items or adaptive assessments, could greatly 
improve the type and amount of instructionally relevant information 
available from the assessments. TEAs have the potential to provide infor- 
mation about what students do or do not know, but also what cognitive 
specific strategies students do or do not apply. 

There will certainly be challenges in the identification of errors and, 
especially, patterns of errors, particularly if we are dependent upon existing 
assessment designs, psychometrics, and data management practices. The 
use of TEAs may prove especially beneficial in overcoming such challenges. 
Research should seek to evaluate the benefit of TEAs in the identification 
of errors and patterns of errors, thus increasing the dynamic nature and 
utility of such assessments. 
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Research should seek out and explicitly investigate TEAs that are 
instructionally useful. These TEAS should be designed to provide instruc- 
tional implications that can be used to guide the learning activities of stu- 
dents at varying levels of knowledge and skills. The research should be 
grounded in sound learning theories and teachers’ instructional experi- 
ences that identify students’ errors as they engage in learning. The erro- 
neous strategies can be identified and documented to provide information 
about where a student is performing along a learning progression, pro- 
viding a valid foundation upon which sound instruction can be based. 

Which Data Matter in TEA? 

TEAs have the potential to acquire, store, process, and communicate a 
wide range of data that can be used to make sound inferences about what 
students know and can do, including complex forms of learning and cog- 
nition. How data are to be used has an important impact on how the data 
should be presented and the level of detail for data collection. For a sum- 
mative assessment, the main inferences typically drawn might be those 
related to the status of the student’s knowledge, skill, and ability at the end 
of a learning experience (e.g., a semester or year of study). In summative 
assessments, data about the learning progressions that students follow to 
learning outcomes may be more difficult to collect given the broad range 
of goals that are being assessed. 

On the other hand, while taking a formative assessment, the main 
inferences typically drawn are those related to what the student is cur- 
rently learning (or not) in the classroom. Formative assessments may pro- 
vide more opportunities to collect data on learning progressions given the 
fewer learning outcomes that are typically assessed in any single formative 
assessment. TEAs may provide multiple tasks that teachers could admin- 
ister formatively and frequently that embody effective instruction. These 
might include a series of probes that could be used to investigate further 
student understanding. Instructional response to findings from formative 
assessments may also follow in real time, more immediately than from 
summative assessments, especially those that are paper-based. If the 
assessments results are intended to inform instruction, it may be a matter 
of great importance what routes have been followed and what patterns of 
errors students have made, because improvements can be made to future 
instruction based on this information. 

Since there may be greater variability in the pathways followed by 
students with disabilities to attain particular learning outcomes, the 
benefits conferred by TEAs may be greater for students with disabilities. 
Furthermore, it may be possible using TEAs to collect data that document 
students’ patterns of reasoning and make more valid inferences about 
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hard-to-measure concepts (e.g., complex problem solving skills in mathe- 
matics and inquiry processes in science). In these hard-to-assess areas, it is 
important to document the patterns of reasoning and errors that students 
make as they engage in multistep, iterative solution strategies. TEAs are 
able to capture this kind of information for the individual student. To use 
such data confidently, it is necessary to validate the inferences. Inferences 
that are important for students with disabilities may vary from or be larger 
than the set of inferences that are important for students without disabili- 
ties, because of, for example, the additional inferences that may need to be 
made about the accessibility of the assessment tasks. 

For summative assessments, TEAs provide increased opportunities 
to gather data about student performance more closely aligned with the 
intended depth of knowledge of the assessment and beyond. TEAs offer 
the opportunity to gather data from student performance using the range 
of tools for designing innovative items and tasks utilizing simulations, 
enhanced graphic design, and flexibility in presenting the tasks in different 
ways to accommodate diverse learners. Some aspects of achievement 
that have proven resistant to all efforts to assess them, such as extended 
writing, mathematical reasoning, and scientific inquiry, may yield to the 
unique range of tools that TEAs make available. These advantages promise 
to add value to the known opportunities technology provides as described 
by Darling-Hammond and Ducommun (2010): 

Technology also organizes data about student learning, enhancing 
system accountability for instruction and reporting by providing 
more efficient, accurate, and timely information to teachers, parents, 
administrators, and policymakers. In the current U.S. context, 
technology can help to integrate information at all levels of the 
system as part of a longitudinal state data system, contributing to a 
rich profile of accomplishment for every student. 

Not only will there be more data, TEA promises richer data that includes 
information about more complex items and tasks (Darling-Hammond & 
Ducommun, 2010). The opportunity to measure students’ learning pro- 
gressions and better understand their cognitive pathways is a significant 
addition to the data available from past assessments. The challenge will lie 
in our ability to analyze the data effectively, once gathered. 
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Research Questions Addressing TEA and Students 
with Disabilities 

As discussed in this article the opportunities provided by TEA to 
enrich knowledge of student cognition, learning, and achievement and 
the impact of disability on learning are many. The challenge lies in a thor- 
ough investigation of the many dimensions of this still-emerging form of 
assessment. This discussion has attempted to briefly review the opportu- 
nities and challenges that TEA provides to help formulate research ques- 
tions that can guide the design, deployment, and validation of TEA in the 
service of students with disabilities. 

Research is needed to pool scientific knowledge across the fields of 
computer-based tools, assessment design, and cognition, as well as instruc- 
tional strategies, for students with and without disabilities, particularly at 
the higher and lower ability levels, and to investigate the benefits and chal- 
lenges of TEAs relative to traditional methods and modes of assessment. 
This article proposes that researchers and experts across disciplinary fields 
in these areas coordinate efforts to address these issues, guided by the 
following research questions regarding the use of TEAs, particularly for 
students with disabilities. 

Measurement of Cognition 

1 . What is known about cognitive pathways and learning progres- 
sions that help in the assessment of students’ content mastery? 

• How are cognitive pathways different or the same for students 
with disabilities? 

• How are learning progressions different or the same for 
students with disabilities? 

• What is the relationship between cognitive pathways and 
learning progressions for students with disabilities? 

• Are cognitive pathways and learning progressions different for 
students with different disabilities? 

• What are the learning progressions for students with different 
types of disabilities in various academic disciplines? 

2. Can students’ positions on the continuum of development and 

learning toward targeted outcomes be determined accurately? 

• What are identifiable benchmarks or milestones that can be 
assessed? 
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3. How can instruction based on evidence-based learning pro- 
gressions be effectively used to better understand the cogni- 
tive pathways appropriate to a student’s abilities and learning 
needs? 

4. Can patterns of errors be identified to provide evidence for 
charting effective learning progressions? Can errors and pat- 
terns of errors be revealed by the following assessment types: 

• Those with a variety of designs and uses, such as paper-pencil, 
online, adaptive, dynamic, formative, summative? 

• Those at various levels of aggregation, such as total sample, 
subgroups, individual students? 

5. If patterns of errors can be identified, what data can be validly 
extracted from those errors and error patterns to benefit the 
following areas? 

• Item and test design and development. 

• Performance assessment development. 

• Score reporting. 

• Cognitive modeling. 

• Production of instructionally relevant and useful data, 
instructional models, remediation, and enrichment. 

• Involvement of students in their own learning. 

• Design of professional development initiatives. 

Interactive-Dynamic Assessment 

1. How can TEA be used to build models of progressive learning 
and performance of students with disabilities? 

• How can the TEA map learning over time? 

• Does adapting and/or scaffolding of assessment items provide 
the instructional information teachers and students need to 
chart learning progressions? How useful is this information to 
teachers in guiding instruction? 

• In what ways could technology improve the diagnostic 
assessment process in a comprehensive assessment system? 
An intervention process? 

2. How do students with disabilities respond to varying assess- 
ment conditions and environments? 
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• How do students respond to the feedback and scaffolding 
procedures inherent in dynamic assessment? 

• What student strategies contribute to correct and incorrect 
responses within a TEA? 

• What is the impact on performance and/or cognition 
of construct-irrelevant variables or barriers (i.e., access, 
navigability, accommodations)? 

3. What effects do interactive-dynamic assessments have on the 
way in which teachers deliver curriculum and instruction to 
students with disabilities? 

Validity Evidence 

1. What data are needed to articulate the validity arguments asso- 
ciated with the intended inferences of technology-enabled for- 
mative and summative assessments, particularly for students 
with disabilities? 

2. How do we expand and improve upon current psychometric 
methods that will be needed in a TEA environment (e.g., com- 
puter adaptive models, multidimensional IRT models, per- 
formance assessment measurement, statistical models for 
simulated environments, additional precision needed at the 
high and low ends of the scale)? 

• Can appropriate psychometric models be developed to 
facilitate diagnostic probing within the context of cognitive 
pathways and learning progressions for students with 
disabilities? 

3 . In what ways do strategies used to design items for static, paper- 
pencil assessments apply to TEA (e.g., accessibility guidelines, 
universal design)? 

4. Can appropriate psychometric models be developed for TEAs 
linked to a learning progression and cognitive pathway for stu- 
dents with disabilities? 

5. What are the constructs, both relevant and/or irrelevant, that 
are measured within the TEA environment? 

6. What inferences can be made with acceptable validity on the 
basis of scores or results of formative and summative tech- 
nology-enabled assessments for students with disabilities? 

• What frameworks and structures are available for framing 
these arguments (e.g., claims, evidence)? 
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• How do these frameworks represent key concepts — -such as 
assessment purpose, the targeted proficiency to be measured 
or fostered, the task situation, characteristics of the students, 
and accessibility barriers — in a way that supports the design 
of assessments that achieve desired quality characteristics 
such as accessibility, validity, learning effectiveness, and 
learning efficiency (Hansen, Zapata-Rivera, & Feng, 2009)? 

• How well do these frameworks address the requirements of 
data from complex constructed responses and data about 
learning? What enhancements to such frameworks are 
needed? 

7. How can empirical research (e.g., analyses of test content, 
students’ cognitive processes, criterion-related evidence) best 
support the claims (and counterclaims) for the intended uses 
of results in the validity argument that has been articulated? 

8. How can Response to Intervention (RTI) techniques (Fuchs 8t 
Fuchs, 1998, 2006) be used effectively to enhance the validity 
of assessment results and subsequent instructional interven- 
tions? 

Consequences of TEA for Students with Disabilities 

1. What benefits can be identified for using TEA strategies with 
children with special needs? Are these benefits different for 
students in the general education curriculum? Benefits may 
include the following: 

• Measuring student performance on the defined construct of 
interest. 

• Measuring aspects of student cognition (i.e. cognitive load, 
problem solving). 

• Linking assessment with improved cognition, instruction, and 
learning. 

- Providing information about effective and less effective 
student cognitive strategies. 

- Providing information to support and improve instruction 
and student learning. 

2. Are there negative consequences for students with disabilities 
using such types of testing? 

• While more precise understandings of points on learning 
progressions will help to guide instruction, is there potential 


J-T-L-A 


Measuring Cognition of Students with Disabilities Using Technology-Enabled Assessments 


33 


for individualized approaches to undermine targeting grade- 
level standards as outlined by IDEA (2004)? 

Designing and Implementing the Research Agenda 

Given the research questions described, designing and implementing 
the research agenda presents a number of challenges. Research questions 
need to be prioritized. Design of the research will require attention to both 
the unique opportunities provided by TEA and to the technical quality 
requirements of any assessment. 

Challenges 

Known challenges should be minimized while unknown challenges 
encountered during the research should be exposed. Measuring students’ 
cognitive strategies in any assessment is a challenge. Some of the chal- 
lenges to assessing various types of students with TEAs include student 
access, school infrastructure and technical capabilities, curriculum, teacher 
preparation, professional development, policy, and others. To respond to 
these challenges, researchers need to: 

• Determine the degree to which TEA enhances measurement of 
students’ cognitive strategies. 

• Clarify the impact of cognitive pathways on learning 
progressions and their role in the design of TEAs for students 
with disabilities. 

• Identify and apply what is already known about the use of 
technology in education. When technology is involved, the 
challenge increases due to uneven use of the tools of technology 
in classroom instruction. 

• Identify generalizable findings. The wide range of disability 
types/categories, both between disability categories and within 
disability categories, will make it difficult to generalize findings 
especially with regard to low-incidence disabilities. The lack of 
consistency across schools and districts in the categorizing of 
students exacerbates this challenge. 

• Conduct rigorous research based on random assignment and 
control groups. This may prove difficult given the many in-place 
programs and regulatory requirements. 

• Identify incentives. While there is a clear need for systematic, 
coordinated, collaborative, and innovative approaches to 
addressing this set of research questions, incentives for engaging 
in such approaches are lacking. 
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• Find or develop the political will to invest the necessary time and 
funding to support systematic, coordinated, collaborative, and 
innovative approaches to yield effective implementations that 
can be brought to scale. 

• Develop and implement training at all levels (e.g., implementers, 
developers, practitioners, and those who train them). 

• Revise practices, policies, and regulations to ensure privacy and 
information protection while enabling a model of assessment 
that includes ongoing student learning data gathering and 
sharing for continuous improvement (Office of Educational 
Technology, 2010). 

Timeframe 

The research should include short- and long-term goals and involve 
both immediate and longitudinal analyses. The research should include 
a combination of observational qualitative research with the use of an 
existing TEA, followed by data analyses over time. 

Immediate Tasks 

Immediate tasks include (1) comprehensive reviews of extant, fugitive, 
and emerging research to determine what is already known about cogni- 
tive pathways relative to differing disabilities; (2) the development of a 
taxonomy of research tasks whereby different researchers could undertake 
different tasks; (3) creation of a catalyst network that facilitates and man- 
ages the synthesis of results across research initiatives; (4) enlistment of 
funding sources and development of political will to the research agenda; 
and (5) rapid prototyping and pilot testing of promising practices. 

Longer-term Tasks 

Longer-term tasks include (1) conducting rigorous research on prom- 
ising practices that have been pilot tested; (2) conducting impact and 
consequential validity studies; (3) bringing promising practices to scale 
and conducting research on the scale-up process/effectiveness; and (4) 
ensuring adequate training for those who will implement various aspects 
of the approaches based on research findings. 

Design 

The design of the TEA should follow standard assessment guidelines 
and meet the technical quality requirements set forth in the Standards 
for Educational and Psychological Tests (AERA, APA, & NCME, 1999) and 
best practice (Downing & Haladyna, 2006). Statistical and qualitative 
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data analyses should be rigorous and appropriate. For example, specific 
measurement models or statistical analyses should be used only when 
the assumptions underlying the models and statistical applications can 
be adequately met. Researchers need to incorporate a research design as 
close to experimental as possible, where students are randomly grouped 
by treatment or control conditions. The design does not necessarily 
require, for example, a comparable paper-based assessment; however 
some ability to differentiate true benefits of the TEA above and beyond 
current approaches (whatever those may be) is important. 

Special attention should be paid to the interaction of student access to 
the assessment and the construct being measured, especially as it impacts 
the performance of students with disabilities. The investigations of access 
should not be limited to accommodations and other access tools. It should 
include how TEA is able to provide access to barrier- free construct-relevant 
performance, which requires greater attention to the target construct and 
how it is measured. For example, examinee response flexibility and the 
capability and flexibility of the TEA to capture data about the construct 
of interest is critical. As such, TEA tools and the research methods should 
attend to influences on, and the interactions between access and construct 
variables (Almond et al., 2010). 

The TEA research design should also incorporate specific methods for 
collecting and rigorously analyzing and testing models of student cogni- 
tion, to include learning progressions and changes in cognition across 
time and task. In either case, the design should capture not only what stu- 
dents know, but also what they do not know and the strategies they use. 

Programmatic evaluation of the instructional and learning environ- 
ment is also an important feature of the design to consider. Understanding 
of the environment is important to the utility of the TEA. For example, a 
TEA may measure the construct perfectly yet be completely unusable in 
the classroom, provide little instructionally relevant information, or be 
unsupported by teachers, technical limits, or competing policy. 

The following steps, in addition to standard test development prac- 
tice, are recommended for addressing the research questions taking into 
account the timeframe of immediate and loner-term tasks suggested 
above. Some of the steps may be undertaken simultaneously. 

1. Identify or conduct reviews of extant, fugitive, and emerging 
research on topics such as the following: 

• Cognitive pathways, with particular attention to students 
with disabilities of various kinds. 

• Learning progressions, with particular attention to students 
with disabilities of various kinds. 
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• Ways in which technology has been used effectively to 
increase access to learning and valid assessment for students 
with various disabilities. 

2. Design and implement research-based pilot tests and perfor- 
mance assessments of learning progression models that are 
grounded in cognitive pathway theory and/or models. Different 
pilot projects would address different types of disabilities. 

• Use the pilot test findings to refine approaches and validate 
the progressions for a variety of students. 

• Try the refined approaches with a different sample of 
students. 

• Refine and test bringing the approach to scale. 

3. Design and pilot test assessments that are informed by the 
learning progression research conducted in step 2. Different 
pilot test projects would address different testing environ- 
ments, such as the following: 

• Formative testing (during units of instruction to allow 
students and teachers to modify learning and instruction). 

• Benchmark/interim testing (between units to monitor within 
year progress toward annual goals). 

• Accountability testing (summative, typically end-of-year to 
determine level of proficiency for policymakers). 

4. Establish the technical quality of the assessments, including 
the extent to which they pinpoint a student’s progress relative 
to a learning progression and benchmark. 

5. Determine how technology can add value to the instructional 
and assessment processes, with the following methods: 

• Identify or design prototypes. 

• Pilot test prototypes. 

• Refine, validate, and try out with another sample. 

• Refine and bring to scale. 
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Documentation 

Finally, as with research on any assessment, plans for gathering evi- 
dence of both the technical quality and consequential validity of TEAs 
should be set forth at the beginning of the research design. The level of 
technical quality required for assessments of students with disabilities, 
when the assessments will be used to make placement or other high- 
stakes decisions, should be at least as high as for students who do not 
have disabilities. In addition to adequate levels of reliability and validity 
for the inferences that will be made using the assessment data, consequen- 
tial validity evidence should be documented and acted upon. It is highly 
desirable that student performance in response to varying interventions 
be documented to monitor progress relative to learning targets and to 
adjust instruction. Such documentation will benefit from user-friendly 
technology-enhanced processes that allow practitioners to quickly docu- 
ment and assess teaching-learning efforts. 


Conclusions 

With the upcoming reauthorization of the Elementary and Secondary 
Education Act (ESEA), an opportunity exists to use TEA to develop assess- 
ments that are not only more inclusive for more students, but also sig- 
nificantly improve the validity of assessment results for all students, 
especially students with disabilities. There is an urgent need to articulate 
a research agenda for TEA that capitalizes on the increased access such 
assessments provide for students with disabilities. The shift to TEA is 
already underway, driven by states’ increasing use of technology to assess 
all students, as educators seek more diagnostic information from their 
assessment system and look for better ways to evaluate skills not easily 
measured on paper-and-pencil assessments. 

TEA holds the promise of particular benefit for students with disabili- 
ties for understanding their cognitive pathways and learning progressions, 
and the ways in which these may be different than those for typical stu- 
dents. There are many potential benefits of using technology for assess- 
ment for all students that have been discussed for more than a decade 
(Bennett, 1995, 2002): 

• Allowing for cost savings and faster reporting when compared 
to traditional modes of assessment (multiple choice and 
constructed response). 

• Potentially minimizing sources of construct-irrelevant variance 
when accommodation or support is provided. 

• Allowing the assessment to be more authentically situated. 
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• Allowing for additional data about student interaction with a 
problem space to be collected. 

• Measuring student use of technology-based tools and resources 
for problem solving in a technology-rich environment that may 
be analogous to how the activity would be completed in the 
classroom and/or real world. 

In summary, there are prototypes that demonstrate the possibilities 
TEA offers to assess a full range of content and skills with interactive- 
dynamic formats that are flexible and can be personalized. Computerized 
adaptive testing (CAT) redesigns educational measurement instruments 
for interactive delivery. Its objective is to select, for each examinee, the 
set of test questions from a precalibrated item bank that most efficiently 
measures that person’s achievement (Thompson & Weiss, 2009). Dynamic 
assessment provides corrective feedback in response to student failure to 
measure both the product and process of learning (Caffrey et al., 2008). 
Scaffolding is a dynamic assessment strategy used to enhance items by 
providing support during learning that are gradually removed when 
learning becomes solidified and/or the learner becomes more indepen- 
dent. Scaffolding in assessment includes structural assistance introduced 
to organize information or guide responses embedded in the presentation 
of the item or task (Perie et al., 2009). Another example of the innovations 
available through technology-enabled assessment is virtual performance 
assessment, where the student experiences an immersive virtual perfor- 
mance assessment, via computer interface, with 3-D virtual contexts, 
digital artifacts, and avatar-based identities that capture what students 
are doing (Clarke-Midura et al., 2010; Mislevy et al., 2003). Requirements 
for assessments in Response to Intervention (Rtl) have motivated the 
development of innovative TEA specifically for students with disabilities. 
Among these are web-based research prototype reading environments that 
include a curriculum-based measurement tool — maze and oral reading flu- 
ency. These dynamic and interactive forms of assessment hold promise for 
improving the range and depth of assessments for all students, especially 
students with disabilities. 

With the need to document the technical quality of these assessments, 
research is needed to establish their validity, especially for students with 
disabilities. Critical research topics include the following: 
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1. Improving measurement of cognition by investigating how the 
cognitive pathways and learning progressions of students with 
disabilities may converge with or differ from those of typical 
students and how error patterns may reveal the challenges 
students with disabilities face in following learning progres- 
sions to achievement of standards. These investigations hold 
promise for using this deepened understanding of the learning 
of students with disabilities to inform teaching strategies. 

2. Use of interactive-dynamic assessments, including gaming 
technology, simulations, collaboration environments, and vir- 
tual worlds, with students with disabilities to help them navi- 
gate the technology environment and access the assessments 
to demonstrate their achievement of complex skills and perfor- 
mances embedded in standards. 

3. Identifying the evidence needed for establishing validity for 
TEA, especially for students with disabilities. This evidence 
may require the development of new psychometric models, 
identifying the data needed to support inferences of TEA, 
minimizing construct irrelevant variance and increasing mea- 
surement precision. The challenge lies in making full use of the 
extensive data that TEA makes it possible to collect about stu- 
dents’ performance. 

Challenges for conducting research on TEA include technology and 
resource limitations, access of researchers to adequate populations to pro- 
duce meaningful findings, and the political will to pursue the next evolu- 
tion in assessment. Rather than continuing down the path of adapting 
traditional assessment strategies, policymakers, assessment providers, 
and other stakeholders will need to marshal the resources and commit- 
ment necessary to fully realize the promise of TEA to make innovation in 
assessment possible. 

The authors offer recommendations for designing a national research 
agenda. Immediate opportunities are provided by expanding the types 
of investigations already underway in interactive-dynamic assessment, 
providing increased access and range through scaffolding and expanded 
access tools, and expanding assessment range and depth through virtual 
performance assessments. TEA makes available a wealth of information 
about cognitive pathways and learning progression. Researchers need to 
take action to fully utilize the opportunities to investigate these promising 
windows into student learning. 

These innovations need validation for application to large-scale sum- 
mative assessments and expanded applications in interim and formative 
assessment. Immediate tasks include the following: 
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1. Comprehensive reviews of extant, fugitive, and emerging 
research to determine what is already known about cognitive 
pathways relative to differing disabilities. 

2. The development of a taxonomy of research tasks whereby dif- 
ferent researchers could undertake different tasks. 

3. Creation of a catalyst network that facilitates and manages the 
synthesis of results across research initiatives. 

4. Enlistment of funding sources and development of political 
will to implement the research agenda. 

5. Rapid prototyping and pilot testing of promising practices. 

Longer-term tasks are as follows: 

1. Conducting rigorous research on promising practices that have 
been pilot tested. 

2. Conducting impact and consequential validity studies. 

3. Bringing promising practices to scale and conducting research 
on the scale-up process/effectiveness. 

4. Ensuring adequate training for those who will implement var- 
ious aspects of the approaches based on research findings. 

Careful attention to documentation of the technical quality, especially 
the validity, of TEA should be set forth at the beginning of the research 
design. Establishing high technical quality for assessments of students 
with disabilities, when the assessments will be used to make placement or 
other high-stakes decisions, should be at least as high a priority as that for 
students who do not have disabilities. 
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